Data Visualization

This week we’ll cover a few packages for visualizing our data:

• ggplot2, our main package for static data visualization with the tidyverse
• timetk, a package built by Matt Dancho for working with time series data
• plotly, an interactive visualization builder that also integrates nicely with ggplot.

library(tidyverse)
library(timetk)
library(plotly)
library(readxl)
library(janitor)
library(lubridate)

knitr::opts_chunk$set(message = FALSE, warning = FALSE, comment = NA)

Let’s load our HPI dataset, wrangle it, and pivot it into long format.

url <- "https://www.fhfa.gov/DataTools/Downloads/Documents/HPI/HPI_PO_monthly_hist.xls"

destfile <- "hpi.xls"

curl::curl_download(url, destfile)

hpi <- read_excel(destfile, skip = 3)

hpi_wrangled <- hpi %>% 
  clean_names() %>% 
  slice(-1) %>%   # remove empty row
  rename(date = month) %>%
  select(date, ends_with("_sa")) %>%  # only keep seasonally adj. data
  # separate() from tidyr package to split date into separate columns for day/month/year 
  separate(date, into = c("year", "month", "day"), sep = '-', convert = TRUE, remove = FALSE) %>% 
  # unite() from tidyr to join columns 
  unite(yr_mon, year, month, sep = "/", remove = FALSE) %>% 
  mutate_if(is.numeric, round, digits = 2) %>%  # round all numeric columns to 2 digits
  # add labels with case_when()
  mutate(season = case_when(between(month, 3, 5) ~ "spring",
                            between(month, 6, 8) ~ "summer",
                            between(month, 9, 11) ~ "fall",
                            # between(month, 12, 2) ~ "winter" == won't work because there's no numbers between 12 and 2
                            TRUE ~ "winter")) %>% 
  select(date, yr_mon, year, month, day, season, everything()) %>%  # reorder columns
  # arrange() from dplyr to sort rows
  arrange(date)

hpi_tidy <- 
  hpi_wrangled %>% 
  select(date, contains("north"), contains("south")) %>% 
  # pivot_longer makes data long, or tidy
  pivot_longer(-date, names_to = "division", values_to = "hpi") %>% 
  group_by(division)

hpi_tidy

ggplot2

And now let’s create a basic ggplot.

ggplot(hpi_tidy, aes(x = date, y = hpi, color = division)) +
  geom_line()

The ggplot2 package uses some unique syntax (the “grammar of graphics”) that allows us to create highly customizable static graphics. This grammar can be a bit hard to grasp, so don’t worry if it takes a while to “click”.

I think about ggplot like this:

• We always start by calling ggplot() to create our plot object. This is (usually!) where we will specify our data and aesthetic mappings, or how our variables should map onto features of the plot like axes, colored groupings, etc.
• Then we add geoms to our plot. This is the step that will actually display our data on the graph. We will always have at least one of these, and sometimes multiple.
• Finally, we can change the appearance of our plot by adding things like themes and changing titles and captions.

Let’s dissect the code above to understand each step.

Wait, why a + instead of %>%? You can think of ggplots in layers. At each step, we’re adding a new layer, like we’re painting on a canvas. This is different than the pipe, which is for passing an object along to a new function.

Faceting

Another thing we can “add” to our ggplots is a faceting layer. Facets divide a plot into subplots based on one of our variables. For example:

ggplot(hpi_tidy, aes(x = date, y = hpi, color = division)) +
  geom_line() +
  facet_wrap(~division)

Scatterplot

There are many kinds of geoms we can add. For example, a scatterplot uses geom_point().

hpi_pct <- 
  hpi_tidy %>% 
  mutate(pct_change = (hpi / lag(hpi)) - 1,
         pct_change_12_mons = (hpi / lag(hpi, 12)) - 1) %>%
  na.omit()

hpi_pct %>% 
ggplot(aes(x = pct_change_12_mons, y = pct_change, color = division)) +
  geom_point() + #alpha = .5
  facet_wrap(~division, ncol = 4) +
  labs(x = "% change (1 mo.)", y = "% change (1 yr.)") +
  theme(legend.position = "none",
        axis.text.x = element_text(angle = 90, hjust = 1, vjust = 0.5),
        plot.title = element_text(hjust = 0.5))

Scatterplot with trend line

We can also add multiple geoms to a ggplot object, e.g. adding a trend line to our scatterplots:

hpi_pct %>% 
ggplot(aes(x = pct_change_12_mons, y = pct_change, color = division)) +
  geom_point() + #alpha = .5
  geom_smooth(method = "lm",  se = TRUE, color = "purple") +
  facet_wrap(~division, ncol = 4) +
  labs(x = "% change (1 mo.)", y = "% change (1 yr.)") +
  theme(legend.position = "none",
        axis.text.x = element_text(angle = 90, hjust = 1, vjust = 0.5),
        plot.title = element_text(hjust = 0.5))

Histogram

There are lots of ways to customize the look and feel of your plots:

hpi_pct %>% 
  ggplot(aes(x = pct_change)) +
  geom_histogram(fill = "darkblue", color = "darkred", bins = 50) +
  facet_wrap(~division) +
  theme(axis.text.x = element_text(angle = 90, hjust = 1, vjust = 0.5)) +
  theme_minimal()

Density

hpi_pct %>% 
  ggplot(aes(x = pct_change)) +
  geom_density(fill = "darkblue", color = "darkred", bins = 50) +
  facet_wrap(~division) +
  theme(axis.text.x = element_text(angle = 90, hjust = 1, vjust = 0.5)) +
  theme_minimal()

Histogram and Density

hpi_pct %>% 
  ggplot(aes(x = pct_change)) +
  geom_histogram(fill = "darkblue", color = "darkred", bins = 20) +
  geom_density(color = "red") +
  facet_wrap(~division) +
  theme(axis.text.x = element_text(angle = 90, hjust = 1, vjust = 0.5))

Shading for recession

We can even combine datasets on the same plot. Let’s create a geom to shade the background of a plot during recession periods.

# First, create our recessions tibble with tribble()
recessions <- 
tribble(
  ~Peak, ~Trough,
  "1948-11-01", "1949-10-01",
  "1953-07-01", "1954-05-01",
  "1957-08-01", "1958-04-01",
  "1960-04-01", "1961-02-01",
  "1969-12-01", "1970-11-01",
  "1973-11-01", "1975-03-01",
  "1980-01-01", "1980-07-01",
  "1981-07-01", "1982-11-01",
  "1990-07-01", "1991-03-01",
  "2001-03-01", "2001-11-01",
  "2007-12-01", "2009-06-01",
  "2020-02-01", "2020-05-01"
  ) %>% 
  mutate(Peak = ymd(Peak),
         Trough = ymd(Trough))


recession_shade <- 
  geom_rect(data = recessions, 
            inherit.aes = F, 
            aes(xmin = Peak, 
                xmax = Trough, 
                ymin = -Inf, 
                ymax = +Inf), 
            fill = 'pink', 
            alpha = 0.5)


hpi_pct %>% 
  ggplot(aes(x = ymd(date), y = pct_change, color = division)) +
  recession_shade +
  geom_line() +
  theme_minimal() +
  theme(axis.text.x = element_text(angle = 90, hjust = 1),
        plot.title = element_text(hjust = 0.5),
        plot.subtitle = element_text(hjust = 0.5),
        plot.caption = element_text(hjust=0)) +
  ylab("") +
  xlab("Percent change, monthly") +
  ggtitle("Housing Price Appreciation", 
          subtitle = "by US Census Division") +
  labs(caption = "data source: FHFA") +
  scale_x_date(limits = c(as.Date(min(hpi_pct$date)), as.Date(max(hpi_pct$date)))) 

timetk for time series data

The timetk package includes a bunch of functions that make working with time series data super easy. This includes functions for easily creating great looking plots of time series data:

hpi_pct %>% 
  ungroup() %>% 
  plot_time_series(date, pct_change, .color_var = division, .smooth = FALSE)

Anomaly diagnostics with timetk

timetk also includes functions for automatic anomaly detection:

hpi_pct %>% 
  filter(division == "south_atlantic_sa") %>% 
  ungroup() %>% 
  plot_anomaly_diagnostics(date, pct_change)

Plotly and ggplotly()

timetk’s interactive plots rely on plotly, a library for building interactive JavaScript visualizations. Plotly is supported in several different languages (including R) and has its own syntax.

Importantly, the plotly R package includes a function called ggplotly() that (you guessed it!) turns ggplots into plotly charts.